Add initial support for frame processor usage directly on tracks#671
Add initial support for frame processor usage directly on tracks#6711egoman wants to merge 6 commits into
Conversation
🦋 Changeset detectedLatest commit: c1e6f43 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| "outDir": "dist", | ||
| "declarationDir": "dist" | ||
| "declarationDir": "dist", | ||
| "lib": ["es2015", "es2021.weakref"] |
There was a problem hiding this comment.
Is this an ok addition to be making here? Or does there need to be a WeakRef polyfill which falls back to some sort of a no-op class wrapper?
(in case it's useful, WeakRef was introduced in node v16.4: mdn link)
|
Going to wait for some of the conversation on the python version of this to shake out before continuing to push this forward. |
There was a problem hiding this comment.
🟡 Missing processor metadata update in localTrackRepublished handler causes stale/empty metadata after reconnection
The PR adds setRoom(this) calls in localTrackPublished, localTrackUnpublished, trackSubscribed, and trackUnsubscribed handlers to push processor metadata. However, the localTrackRepublished handler (fired during full reconnect when the SDK auto-republishes tracks) has no corresponding setRoom call or metadata push. After republish, the publication receives a new SID via publication.updateInfo(newInfo) at room.ts:567, but the track's info.sid (used by pushProcessorMetadataToStream at track.ts:125) is never updated. This means any subsequent call to pushProcessorMetadataToStream (e.g., when a new AudioStream is registered on the track after reconnection) will fail the lookup publication.sid === trackSid at track.ts:141 because the publication now has the new SID while the track still has the old one. The processor receives empty participantIdentity and publicationSid.
(Refers to lines 566-570)
Prompt for agents
In the localTrackRepublished handler in room.ts (around line 561-573), after the publication info is updated and re-keyed in the map, the track's processors are never notified of the new publication SID. Two things need to happen:
1. The track's own info.sid needs to be updated to match the new publication SID, because pushProcessorMetadataToStream in track.ts uses this.sid (which reads this.info?.sid) to look up the publication in the trackPublications map.
2. A metadata push should be triggered so that processors attached to existing AudioStreams on this track receive the updated stream info (new publicationSid).
A fix would be to add something like this inside the `if (publication)` block in the localTrackRepublished handler, after the publication is re-keyed:
if (publication.track) {
publication.track.info = { ...publication.track.info, sid: publication.sid }; // or however TrackInfo is updated
publication.track.setRoom(this); // re-pushes metadata to all registered audio streams
}
Note: setRoom(this) when oldRoom === room will still properly re-register the listener and push metadata to streams. However, be careful that the track's info structure is updated correctly (it's a protobuf TrackInfo object).
Was this helpful? React with 👍 or 👎 to provide feedback.
Updates the node sdk so that
FrameProcessor-based noise cancellation providers can be used directly onAudioStream, without having to go through the agent's RoomIO to be able to initialize itself with credentials.For example, with this change, something like the below becomes possible:
The way this works -
Tracks now keep track of which room they are part of (holding aWeakRefvalue). When the room a track is in changes, it computes new frame processor options and sends these to anyAudioStreams which are associated with the track.The
noiseCancellationLeaveOpenparameter allows the agents sdk to construct anAudioStreamwith a frame processor which remains open across the whole session, and won't be auto-closed when the track is closed.Todo